Add ability to enforce concurrent query limits#3257
Add ability to enforce concurrent query limits#3257lbschanno wants to merge 51 commits intointegrationfrom
Conversation
Add the ability to enforce concurrent query limits across a group of webservers. Zookeeper is used to track active queries and the following data: - The query ID - The user who submitted the query - The system the query was submitted on - The query logic the query originated from When the `ActiveQueryTracker` is instructed to track a query, the following nodes will be created in Zookeeper under the 'ActiveQueries' namespace: ``` /users/<userDn>/<queryId> /systems/<systemName>/<queryId> /queryLogics/<queryLogic>/<queryId> /queries/<queryId> /queries/<queryId>/user [data = byte[] value of userDn] /queries/<queryId>/system [data = byte[] value of systemName] /queries/<queryId>/queryLogic [data = byte[] value of queryLogic] /queries/<queryId>/heartbeats ``` This is done through the use of the `ActiveQueryTracker` class. In addition to managing the nodes that record information about the query, the `ActiveQueryTracker` class is also responsible for providing instances of the `QueryHeartbeat` class. A `QueryHeartbeat` is a wrapper around an ephemeral PersistentNode, provided by the Apache Curator library. As long as this node is present in Zookeeper for a particular query, the query will be considered to be active. Should the webservers fail over and the Zookeeper connection drop, these heartbeat nodes will automatically be deleted by Zookeeper. The `ActiveQueryTracker` is also responsible for providing instances of the `ActiveQuerySnapshot` class, which represent a snapshot of total active queries at a point in time that are associated with a particular user, system, or query logic. Query limit enforcement is done through the `QueryLimiter` class. Given a user, system, and query logic, it can determine if any of the following limits have been exceeded: - The max allowed concurrent queries for the user. - The max allowed concurrent queries of the query logic for the user. - The max allowed concurrent queries for the system. - The max allowed concurrent queries of the query logic for the system. Limits may be defined and customized on a per-user and per-system basis. They may also be defined for groups of query logics. The classes `UserLimitProvider`, `SystemLimitProvider`, and `QueryLogicGroupLimitProvider` are respectively responsible for identifying the best limits to enforce for a user, system, and query logic. They will be initialized in the `QueryLimiter` after providing a `QueryLimitConfiguration` instance. The following can be configured: On a system-wide basis: - The default concurrent user query limit. This applies to the total number of queries a user may run across all systems. May be overridden per user. - The default concurrent system query limit. Primarily to avoid a system getting overloaded. May be overridden per system. - The default of whether queries submitted to a system are counted towards the user's concurrent query total. This is always true. On a per-system basis: - The system name/ids the configuration targets. Regex matching is supported. - The concurrent system query limit. Overrides the system-wide value. - Whether queries submitted to the system count towards a user's concurrent query total. Overrides the system-wide value. - The concurrent system query limit for different query logic groups. Regex matching against group names is supported. on a per-user basis: - The user DN. - The user's concurrent query limit. Overrides the system-wide configuration. - The user's concurrent query limit for different query logic groups. Regex matching against group names is supported. On a per-query-logic-group basis: - The group name. - The query logics included in the group. Regex matching is supported. - The default concurrent user query limit. This applies to the total concurrent queries a user may run that originate from a query logic in the group across all systems. Given the possibilities for exact matches, partial regex matches, and wildcard regex matches, the determination of the best limit to use for any particular system or query logic is done by sorting matches into the following 'matching buckets' (in best-match priority): 1. Exact match 2. Partial regex (non-wildcard-only) 3. Wildcard-only regex and then selecting the lowest limit from the best bucket where we first found a match. Currently the `QueryLimiter` is used in `QueryExecutorBean`, along with a `QueryHeartbeatCache` instance to cache heartbeats and keep them alive when a running query is cached for retrieval later. For the purposes of this feature, a query is considered to start when an Accumulo connection is retrieved from the connection factory, and is considered to end when the connection is returned to the factory. The following error codes have been added: 412-20 - Concurrent query limit exceeded 500-164 - Error checking concurrent query limits Closes #3100
156ed51 to
9c59e98
Compare
…eryLimiter in QueryLimiterFactory.xml, remove scope attribute on queryLimiter and queryLimitConfiguration
...vices/cached-results/src/main/java/datawave/webservice/results/cached/CachedResultsBean.java
Outdated
Show resolved
Hide resolved
web-services/query/src/main/java/datawave/webservice/query/limit/ActiveQueryTracker.java
Outdated
Show resolved
Hide resolved
Co-authored-by: foster33 <84727868+foster33@users.noreply.github.com>
Co-authored-by: foster33 <84727868+foster33@users.noreply.github.com>
web-services/query/src/main/java/datawave/webservice/query/limit/ActiveQueryTracker.java
Show resolved
Hide resolved
ivakegg
left a comment
There was a problem hiding this comment.
I like what I am seeing in here. I am thinking about potential error scenarios where things could get out of sync. Currently we have the following maps related to queries:
- QueryCache: A map of query id to RunningQuery instances. Used by QueryExpirationBean to check for expired queries.
- HeartbeatCache: A map of query id to heartbeat.
It might be worth adding a loop in the QueryExpirationBean that goes through the ids in the HeartbeatCache to verify that those queries are still running per the QueryCache. This would help to prevent anything getting out of sync if we have some unknown error cases that are not being accounted for. Other than that I am good to start integration testing this.
|
@ivakegg understood, I'll add in a loop to the QueryExpirationBean for synchronization with the heartbeat cache. |
|
Added synchronization safeguard. |
|
So I got this up and running and watching the zookeeper entries I realized that there has been one requirements which was not translated correctly and I apologize for not realizing this earlier. The "system" limit is supposed to be base on the "systemFrom" query parameter and not the hostname on which the query is running. I will take a shot at modifying the code accordingly. |
Configuration
Add the ability to enforce concurrent query limits across a group of webservers. Zookeeper is used to track active queries and the following data:
Query limit enforcement is done through the
QueryLimiterclass. Given a user, system, and query logic, it can determine if any of the following limits have been exceeded:Limits may be defined and customized on a per-user and per-system basis. They may also be defined for groups of query logics. The classes
UserLimitProvider,SystemLimitProvider, andQueryLogicGroupLimitProviderare respectively responsible for identifying the best limits to enforce for a user, system, and query logic. They will be initialized in theQueryLimiterafter providing aQueryLimitConfigurationinstance. The following can be configured:On a system-wide basis:
On a per-system basis:
On a per-user basis:
On a per-query-logic-group basis:
Given the possibilities for exact matches, partial regex matches, and wildcard regex matches, the determination of the best limit to use for any particular system, query logic, or query logic group is done by sorting matches into the following 'matching buckets' (in best-match priority):
Implementation
Checking limits and marking as active/inactive is done through the
QueryLimiterclass. The three main methods to know are:QueryLimiter.checkForLimits()QueryLimiter.countQueryTowardsLimits()QueryLimiter.stopCountingQueryTowardsLimits()When a query is marked as active via
QueryLimiter.countQueryTowardsLimits(), it will delegate to the ActiveQueryTracker, which will in turn create nodes in Zookeeper under theActiveQueriesnamespace. WhenActiveQueryTracker.trackQuery()is called, the following nodes are created:ActiveQueryTracker.trackQuery()will return aQueryHeartbeatthat contain a list ofPersistentNode(provided by the Apache Curator library) wrappers around the ephemeral nodes listed above. TheQueryHeartbeatwill maintain the connection to Zookeeper and attempt to keep the ephemeral nodes present in Zookeeper untilQueryHeartbeat.stop()is called. IfQueryHeartbeat.stop()is called, or the webserver crashes, the ephemeral nodes will automatically be deleted by Zookeeper.The following error codes have been added:
Closes #3100